feat: recover terminal sessions after worker restart - #15
Merged
Conversation
Introduce TerminalSessionRecoveryCandidate/Result/Report/Ack messages so workers can reconcile terminal sessions with Console after reconnect.
Send recovery candidates in ConnectAck, apply the worker recovery report to refresh leases and drop missing sessions, and gate dispatch until recovery completes. Mark routes unavailable instead of clearing them on disconnect so the bound session_id keeps returning 503 session_unavailable while the worker is offline or reconciling.
Use deterministic container names and schema/session-hash labels so terminal containers survive worker exit. On reconnect, reconcile Console candidates before accepting commands: restart stopped containers, restore the exact lease, and remove local orphans Console no longer recognizes. PreserveOnClose keeps containers running on graceful shutdown.
Tag terminal sandboxes with session-id metadata, add E2B List/Connect helpers to rediscover paused sandboxes, and reconcile Console recovery candidates on reconnect before accepting commands. Restarted sandboxes keep their original session_id and lease.
Derive deterministic box names from session-id hashes so boxes survive worker exit, and reconcile Console recovery candidates on reconnect before accepting commands. Restarted boxes keep their original session_id and lease.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add terminal session recovery so live sessions survive a worker restart. On reconnect the worker reconciles its local state with Console's recovery candidates, restores session leases, and only then starts accepting commands.
Changes
API / Proto
TerminalSessionRecoveryCandidate/Result/Report/Ackproto messages for the reconciliation handshakeConsole
ConnectAck; apply the worker's recovery report to refresh leases and drop missing sessionssession_unavailable(503) on disconnect instead of clearing them, so bound session IDs return a clean error while the worker is offline or reconcilingworker-docker
PreserveOnClosekeeps containers running on graceful shutdownworker-bridge-e2b
worker-boxlite